This invention relates to digital imaging and image processing techniques, and more particularly relates to digital imaging of low-light-level scenes at real time video image display speeds.
Imaging of low-light-level environments is an important capability for enabling military and law enforcement surveillance, aviation and automotive navigation, and for numerous industrial and consumer manufacturing and production processes, among other applications. For example, the conventional low-light-level imaging systems known as so-called night vision scopes are routinely employed as a principal means for enabling night time mobility and navigation by military personnel wearing helmet-mounted scopes while traveling on foot or navigating in a vehicle, e.g., a jeep, truck, helicopter or jet.
Historically, low-light-level imaging systems, including conventional night vision scopes, have been based on use of an electro-optic image sensor that provides a gain mechanism for amplifying ambient input light to produce an output image signal level that is adequate for visual display, e.g., either by a direct view display or, with the addition of an electronic imaging system, a remote view display. For example, in a typical electro-optic system such as a so-called intensifier tube imager, a lens is used to focus ambient light, such as moonlight or starlight reflected off of a scene, onto a photocathode in a vacuum tube, the photocathode being sensitive to, e.g., light from the yellow through near-infrared portion of the electromagnetic spectrum. Under application of a high voltage between the photocathode and a micro-channel plate also located in the vacuum tube, an input ambient photon incident on the photocathode causes emission of a single electron from the photocathode and acceleration of the electron toward the micro-channel plate. Upon striking the micro-channel plate, the single electron creates a cascade of many electrons, which together are accelerated toward a phosphor screen by a second applied voltage. The kinetic energy of the electrons striking the phosphor causes the phosphor to glow. This electron cascade mechanism results in amplification of the input ambient light, typically by about four orders of magnitude, to produce a visible image on the phosphor screen. The phosphor image exists only momentarily in the glow of the phosphor screen, and does not exist in a storable or readable form.
To produce a real-time electronic video image, which typically is set at a frame rate of about 30 frames/second, based on the optical images formed by an intensifier tube on a phosphor screen, it is common practice to optically couple the phosphor screen to a conventional charge-coupled-device (CCD) electronic imaging camera. Optical coupling is typically achieved using an intermediate coupling lens or an optical fiber taper bonded to the CCD and either bonded or integrally-connected to the intensifier tube phosphor screen. The resulting low-light video camera, or so-called intensified-CCD camera, relies entirely on the cascade gain mechanism of the intensifier tube to provide a phosphor image that is adequately amplified to be sensed by the conventional CCD imager.
Intensified-CCD cameras like the one described above, while capable of producing a real time low-light-level video sequence, have historically been severely restricted with regard to other performance criteria. In particular, the intra-scene dynamic range of an image produced by an intensified-CCD imaging system is severely limited by the image intensifier tube. Furthermore, image resolution is severely degraded at low light levels due to electronic noise associated with the intensifier tube cascading gain mechanism. This electronic noise adds background image intensity noise and can even xe2x80x9cswampxe2x80x9d low intensity images, resulting in an output image that is a poor rendition of the imaged scene. Intensifier tube imagers not including a CCD electronic camera are of course also subject to the resolution and intra-scene dynamic range limitations imposed by the intensifier tube gain mechanism.
These limitations are exacerbated in imaging low-light-level scenes because the same scene may contain very low brightness areas as well as dramatic intra-scene intensity fluctuations due, e.g., to man-made light. But because intensifier tube imagers and intensified-CCD imaging systems intrinsically rely on the vacuum tube cascading gain mechanism for production of a viable phosphor image, the dynamic range limitation imposed by the gain mechanism must be accepted, resulting in either loss of darker areas in the scene or excessive blooming in the brighter areas of the scene. Blooming is here meant as a localized brightness saturation that spills over to other nearby areas. As a result, intensifier tube imagers and intensified-CCD imaging systems are restricted to relatively small dynamic range imaging; typically no more than about 200 gray levels can be enabled by even the best vacuum tube-based systems, and generally, far fewer gray levels span the restricted intra-scene dynamic range.
Additional inherent limitations of intensifier vacuum tube technology limit the overall performance of intensifier tube imagers and intensified-CCD imaging systems. For example, the finite time required for a phosphor image produced by an intensifier tube to dissipate from the phosphor results in deleterious image artifacts in a temporal sequence of images when there is motion in the scene. In addition, vacuum tubes, being formed of glass, are fragile, and therefore require special handling considerations for the image system in which they are incorporated. The photocathode and micro-channel plate used in the intensifier tube have relatively short life cycles, requiring frequent replacement and repair. Furthermore, vacuum tubes are relatively large in size, limiting the minimum overall imaging system size. Vacuum tubes are also relatively expensive, adding significant cost to the overall cost of the imaging system. Many other performance, handling, and packaging limitations are additionally imposed by the intensifier vacuum tube technology.
Another class of low-light-level imaging systems, known as so-called slow-scan or frame-integrating cameras, do not rely on a gain mechanism to amplify ambient light for producing a viable electronic image. Instead, a slow-scan camera typically includes only a CCD imager that in operation is exposed to a low-light-level scene for an extended period of time, i.e., seconds or longer, during which time the device accumulates a large number of photoelectrons; after a time period sufficient to accumulate an adequate photoelectron count, a viable electronic image can be produced by the CCD camera. This technique overcomes some of the practical limitations of intensifier tube imagers and intensified-CCD image systems, but is inherently limited to extremely slow image capture speeds. Slow-scan cameras thus cannot accommodate real time imaging for production of video sequences at rates even close to about 25-30 frames per second. Indeed, slow-scan cameras are typically used in applications that are not primarily time-sensitive; for example, being used in astronomical applications as an electronic substitute for long-exposure film photography through telescopes.
Many critical low-light-level surveillance and mobility applications require real time digital video imaging of night time scenes, e.g., air-land scenes, characterized by a large intra-scene dynamic range. But like the systems described above, conventional low-light-level imaging systems developed heretofore have provided only suboptimal performance under such complex conditions and are further restricted by additional performance and practical limitations that impede or inhibit a high level of operational performance in applications for which real time low-light-level imaging is critical.
The invention overcomes limitations of past low-light-level imagers to provide an imaging system that attains superior performance at real time speeds. In a first aspect, the invention provides an imaging system for imaging a scene to produce a sequence of image frames of the scene at a frame rate, R. The imaging system includes an optical input port for accepting input light from the scene and a charge-coupled imaging device including pixels configured in a charge storage medium, the charge-coupled imaging device located in relation to the input port such that input light from the scene impinges device pixels. The charge-coupled imaging device produces an electrical pixel signal of analog pixel values based on the input light. In the invention, an analog signal processor is connected to the charge-coupled imaging device for amplifying the pixel signal, and an analog-to-digital processor is connected to the analog signal processor for digitizing the amplified pixel signal to produce a digital image signal formatted as a sequence of image frames each of a plurality of digital pixel values and having a dynamic range of digital pixel values represented by a number of digital bits, B. Finally, the imaging system includes a digital image processor connected to the analog-to-digital processor for processing digital pixel values in the sequence of image frames to remap the dynamic range of the frames to a compressed dynamic range of remapped pixel values represented by a number of digital bits, D, where D is less than the number, B. A sequence of output image frames of remapped pixel values representative of the imaged scene are thereby produced at the frame rate, R, with a latency time of no more than about 1/R.
In preferred embodiments, the imaging system includes a display connected to receive the output image frame sequence and to display the sequence at the frame rate, R. Preferably, the digital image processor of the imaging system consists of a center-surround-shunt processor for adaptively enhancing contrast of digital pixel values from the analog-to-digital processor based on values of neighboring pixels in an image frame, and for adaptively normalizing the enhanced pixel values such that the enhanced pixel values are within a compressed and normalized dynamic range. The digital image processor further preferably includes a statistics processor for acquiring pixel value statistics about the digital pixel values from the analog-to-digital processor and for acquiring pixel value statistics about the enhanced and normalized pixel values from the center-surround-shunt processor, and a remapping function processor for constructing a pixel value remapping function based on the pixel value statistics acquired by the statistics processor, the remapping function constituting a rule for remapping the enhanced and normalized pixel values from the center-surround-shunt processor to a selected output dynamic range represented by the number of digital bits D. The digital image processor further preferably includes a remap processor for applying the remapping function from the remapping function processor to the enhanced and normalized pixel values from the center-surround-shunt processor to produce a sequence of output image frames of remapped pixel values representative of the imaged scene. Preferably, the remapping function processor constructs a pixel value remapping function for a given image frame before pixel values in that frame are processed by the center-surround-shunt processor.
In other preferred embodiments, the number of digital bits, B, representing the dynamic range of digital pixel values produced by the analog-to-digital processor is greater than 8 and the number of digital bits, D, representing the compressed dynamic range of remapped pixel values is no larger than 8; preferably the frame rate, R, is at least about 25 frames per second.
In another aspect, the invention provides an imaging system for imaging a scene to produce a sequence of image frames of the scene at a frame rate, R, of at least about 25 image frames per second. The imaging system includes an optical input port for accepting input light from the scene and a charge-coupled imaging device having an array of pixels configured in a charge storage medium. The charge-coupled imaging device is located in relation to the input port such that input light from the scene impinges device pixels, such that the charge-coupled imaging device produces an electrical pixel signal of analog pixel values based on the input light. An analog signal processor is connected to the charge-coupled imaging device for amplifying the pixel signal, and an analog-to-digital processor is connected to the analog signal processor for digitizing the amplified pixel signal to produce a digital image signal formatted as a sequence of image frames each of a plurality of digital pixel values and having a dynamic range of digital pixel values represented by a number of digital bits, B, where B is greater than 8. A digital image processor is connected to the analog-to-digital processor for processing digital pixel values in the sequence of image frames to produce an output image frame sequence at the frame rate, R, representative of the imaged scene, with a latency of no more than about 1/R.
In the invention, the output image frame sequence is characterized by noise-limited resolution of at least a minimum number, NM, of line pairs per millimeter, referred to the charge-coupled imaging device pixel array, in an imaged scene as a function of illuminance of the input light impinging the charge-coupled imaging device pixels. For a scene characterized by a contrast of about 0.3, for a human observation time of about 0.05 seconds, and for an image scene frame rate of about 30 frames per second, NM is given as NM=1900 L0.51, where L is the value of illuminance of the input light impinging the charge-coupled imaging device pixels, for at least one illuminance value between a range of illuminance values of about 1xc3x9710xe2x88x922 LUX and 5xc3x9710xe2x88x927 LUX.
In preferred embodiments of the invention, L, the illuminance of the input light impinging the charge-coupled imaging device pixels, ranges between about 1xc3x9710xe2x88x923 LUX and 1xc3x9710xe2x88x926 LUX. Preferably, the frame rate, R, is at least about 30 frames per second. In preferred embodiments, the optical input port comprises a lens, and a display is connected to receive the output image frame sequence and to display the sequence at the frame rate, R. In other embodiments, the digital image processor and the display are in communication with but located remote from the charge-coupled imaging device and the analog-to-digital processor; alternatively, the charge-coupled imaging device, the analog-to-digital processor, and the display are in communication with but located remote from the digital image processor and a power supply. In other embodiments, a communication link is connected to the digital image processor for transmitting to a remotely located receiver the sequence of output image frames; a user controller is preferably included for controlling the frame rate, R, and resolution of the imaging system, within operational limits of the imaging system.
In other embodiments, the charge-coupled imaging device pixels are configured in a charge storage substrate having a front side supporting pixel interconnections and a back side having no substantial topology, with buried channels in the substrate defining charge packet storage wells for the pixels. Preferably the charge-coupled imaging device is located in relation to the optical input port such that input light from the scene impinges the back side of the substrate. In other preferred embodiments, the substrate is a silicon substrate characterized by a resistivity of at least about 1000 xcexa9-cm.
Preferably, the electrical pixel signal of pixel values has a dynamic range of at least about 1000 distinct pixel value levels and most preferably a dynamic range of at least about 3000 distinct pixel value levels.
In other embodiments, the configuration of pixels in the charge storage substrate is configured as an imaging pixel array on which input light impinges to produce charge packets in the buried channels of the imaging array pixels and a frame storage pixel array shielded from impinging input light, charge packets in the imaging pixel array being transferred to the frame storage pixel array for producing an electrical pixel signal of analog pixel values based on the input light. Preferably, the imaging pixel array consists of an array of electronically-shuttered pixels, each pixel in the array being selectively electronically controllable by the pixel interconnections to inhibit storage of charge packets in the buried channel of that pixel while input light impinges the charge-coupled imaging device.
In other preferred embodiments, the pixel interconnections supported on the front side of the charge storage medium define a three-phase clocking configuration for transferring charge packets in the buried channels; and the pixel interconnections supported on the front side of the charge storage substrate provide interconnections for selective electronic transfer of a charge packet in a given pixel of the imaging pixel array to an adjacent pixel located in a different row of the imaging pixel array, and provide interconnections for selective electronic transfer of a charge packet in a given pixel in the imaging array to an adjacent pixel located in a different column of the imaging pixel array. In other embodiments, charge packets resident in a portion of the pixels in the frame storage pixel array are summed prior to production of an electrical pixel signal of analog pixel values.
In other preferred embodiments, the pixels of the charge-coupled imaging device are configured in a charge storage medium comprising a substrate that is curved in a selected nonplanar focal surface profile and located a selected distance from the lens with the focal surface facing the lens, the focal surface profile and lens-to-substrate distance selected such that the input light is in focus at the location of the substrate. Preferably, a cooling device is provided in contact with the charge storage substrate to suppress dark current charge packet generation in pixels of the charge-coupled imaging device. The cooling device preferably consists of a thermo-electric cooling device.
In another aspect, the invention provides an imaging system having optical input port for accepting input light from the scene, a first charge-coupled imaging device having pixels configured in a charge storage medium and located in relation to the input port such that at least a central field-of-view region of the input light impinges device pixels to produce an electrical central field-of-view pixel signal of analog pixel values based on the input light, and a second charge-coupled imaging device having pixels configured in a charge storage medium and located in relation to the input port such that at least a peripheral field-of-view region of the input light impinges device pixels to produce an electrical peripheral field-of-view pixel signal of analog pixel values based on the input light. An image processor is connected to receive the central field-of-view pixel values and peripheral field-of-view pixel values to amplify and digitize the pixel values and to blend the central field-of-view pixel values with the peripheral field-of-view pixel values to produce a sequence of composite image frames, each composite image frame having digital central field-of-view pixel values in a central region of the composite image frame and having digital peripheral field-of-view pixel values surrounding the central region to form a peripheral region of the composite image frame. The sequence of composite image frames is produced at the frame rate, R, with a latency time of no more than about 1/R.
In preferred embodiments, a display is connected to receive the sequence of composite image frames and display the sequence at the frame rate, R, with the central image region of each composite image frame displayed at unity magnification and the peripheral image region of each composite image frame displayed at a magnification less than unity. Preferably, the display subtends a field of view, with respect to a display viewer, that exceeds the field of view subtended by the central image region, and preferably, the central image region subtends an angle of at least about 30 degrees and the peripheral image region subtends an angle of at least about 80 degrees.
In preferred embodiments, the system includes a field-of-view separator aligned with the optical input port for directing the central field-of-view region of the input light to the first charge coupled imaging device and for directing the peripheral field-of-view region of the input light to the second charge coupled imaging device; the system also preferably includes a long focal length lens located between the field-of-view separator and the first charge-coupled imaging device to focus the central field-of-view region of the input light onto the first charge-coupled imaging device; and a short focal length lens located between the field-of-view separator and the second charge-coupled imaging device to focus the peripheral field-of-view region of the input light onto the second charge-coupled imaging device.
The invention provides, in another aspect, a charge-coupled imaging device for imaging a wide field-of-view scene to produce a sequence of image frames of the scene at an image frame rate, R, the imaging device consists of a short focal length lens for accepting light from the scene to be imaged and a charge storage medium consisting of a charge storage substrate having a front side and a back side, the charge storage substrate being curved in a selected nonplanar focal surface profile and located a selected distance from the lens with the focal surface facing the lens, the focal surface profile and lens-to-substrate distance selected such that the light accepted by the lens is in focus at the position of the substrate. A support substrate is provided on which the nonplanar charge storage substrate is supported to maintain the selected surface profile of the charge storage substrate. An array of pixels is defined in the charge storage substrate by pixel interconnections supported on the front side of the substrate, exposure of the substrate to light from the scene through the lens producing charge packets in the pixels, the pixel interconnections providing selective electronic temporal control of transfer of charge packets from one pixel to another in the substrate. Means for suppressing generation of dark current charge packet generation in the substrate pixels is provided, as well as an output circuit for converting the charge packets in the pixels to an electrical pixel signal of output pixel values based on the light from the scene, a plurality of pixel values together forming an image frame, the output pixel values being produced at a rate corresponding to the image frame rate, R.
The invention further provides a charge-coupled imaging device consisting of a charge storage medium comprising a substrate having a front side and a backside, pixels defined in an array of pixel rows and pixel columns in the charge storage substrate by a plurality of columns of buried channels in the substrate and by a plurality of rows of pixel gates on the substrate over the buried channel columns. A channel stop region is provided in the substrate between and at the periphery of the columns of buried channels, and a serial output register is defined by a row of register gates on the substrate over the buried channels and adjacent to the last pixel row in the pixel array. The output register includes an output stage defined by an output stage gate at the end of the register gate row and a corresponding output stage buried channel comprising an end column of the plurality of buried channel columns, the output stage buried channel extending to the output stage gate. A charge-collection junction is provided adjacent to the output stage gate and defined by a p/n junction in the substrate for collecting charge generated in the array of pixels and output at the output stage gate.
An output stage charge funnel is located between the output stage gate and the charge-collection junction for funneling charge in the output stage buried channel to the charge-collection junction. The output stage charge funnel is defined by a buried implant having a first width at the end of the buried channel and a second width at the charge-collection junction, the first width being larger than the second width.
Preferably, an output circuit is connected to the charge-collection junction for converting the collected charge to an electronic representation of a scene being imaged, the output circuit including a capacitor, an output transistor, and a reset transistor all located in the substrate. The capacitor is connected to the charge-collection junction for producing a voltage corresponding to a given amount of charge collected at the charge-collection junction. The reset transistor is defined by a reset gate and a reset buried channel in the substrate and extends between a reset transistor bias contact and the reset gate. A charge funnel is located between the reset gate and the charge-collection junction for draining charge from the output stage capacitor after the output circuit has produced an electronic representation for a given amount of charge collected at the charge-collection junction. The reset charge funnel is defined by a buried implant having a first width at the end of the reset buried channel and a second width at the charge-collection junction, the first width being larger than the second width. Preferably, the charge-collection junction consists of a dopant region in the substrate, dopant in the region provided by diffusion of dopant from a doped conducting layer deposited over the charge-collection junction location into the substrate under the conducting layer at the charge-collection junction location.